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Background 

The  primary  objective  o£  this  study  was  to  obtain  performance 
capability  measures  for  various  classes  of  clinical  laboratories  in  the 
United  States  and  to  determine  if  there  are  significant  differences  in 
analytical  accuracy  which  would  warrant  remedial  action  by  public  agencies 
or  tlie  private  sector. 

The  procedure  utilized  was  to:     (1)  establish  a  Scientific  Advisory 
Committee  of  governmental  and  health  industry  representatives,  (2)  develop 
a  survey  design,  (3)  contact  professional  and  regulatory  groups  to  solicit 
laboratory  participation,  (4)  procure  and  distribute  two  sets  of  laboratory 
samples  in  clinical  chemistry,  hematology  and  bacteriology  to  laboratories 
participating  in  the  study,  and  (5)  statistically  analyze  the  results 
reported  by  the  laboratories. 


Participants 

Six  types  or  groups  of  clinical  laboratories  participated  in  the 
study: 

1.  Interstate — laboratories  engaged  in  interstate  commerce  and 
licensed  by  the  Center  for  Disease  Control,  Health  Services  and  Mental 
Health  Administration,  under  the  Clinical  Laboratory  Improvement  Act 

of  1967  (CLIA  '67)  in  one  or  more  of  the  specialties  under  consideration 
in  this  study. 

2.  American  Academy  of  Family  Physicians — ^private  physician 
laboratories,  generally  small,  which  are  currently  affiliated  with  AAFP. 

3.  American  Society  of  Internal  Medicine — ^laboratories  operated  in 
conjunction  with  a  private  physician's  practice  which,  like  AAFP  affiliated 
laboratories,  are  presently  exempt  from  any  Federal  licensure  program. 

4.  Joint  Commission  on  Accreditation  of  Hospitals — laboratories 
within  hospitals  which  are  accredited  under  JCAH. 

5.  Medicare  Certified  Hospitals — ^laboratories  in  hospitals  which 
are  Medicare  providers  under  Title  XVIII,  Health  Insurance  for  the  Aged. 
Hospitals  and  laboratories  accredited  under  JCAH  were  excluded  from  this 
category. 

6.  Medicare  Certified  Independent — private  and  commercial  laboratories 
which  are  reimbursed  for  certain  laboratory  procedures  under  Medicare,  but 
which  are  not  normally  licensed  under  CLIA  '67  or  accredited  by  JCAH. 
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Approximately  1,000  laboratories  participated  in  the  study.  The 
number  within  a  category  ranged  from  43  (AAFP)  to  231  (Interstate) .  A 
seventh  category  of  18  reference  laboratories  served  as  a  control  group. 

Methodology 

Separate  proficiency  test  specimens  were  prepared  for  clinical  chemistry, 
hematology  and  microbiology.    Criteria  for  selection  of  the  clinical 
chemistry  and  hematology  constituents  to  be  analyzed  were  that  laboratory 
analysis  should  be  routine,  and  that  fairly  well  developed  analytical 
procedures  exist.    Each  clinical  chemistry  shipment  included  normal  and 
abnormal  samples.    The  laboratories  were  asked  to  determine  the  concentration 
of  eight  constituents:    glucose,  urea  nitrogen,  calcium,  total  bilirubin, 
cholesterol,  uric  acid,  sodium,  and  total  protein.    Each  hematology  ship- 
ment required  analysis  of  red  blood  count,  white  blood  count,  hemoglobin, 
hematocrit,  and  mean  corpuscular  volume  levels  in  both  normal  and  abnormal 
specimens.    Five  pure  cultures  of  ordinary  bacteria  were  used  as  sample 
cultures  for  identification  in  the  microbiological  portion  of  the  study. 

Two  shipments  of  specimens  and/or  cultures  were  sent  to  each 
participating  laboratory.    All  analytic  results  were  reported  by  the 
laboratory  on  forms  shipped  with  the  test  specimens. 


Results  -  Clinical  Chemistry 

There  were  no  significant  differences  (at  the  90%  confidence  level) 
among  the  average  laboratory  results  obtained  by  the  groups  participating 
in  the  study.    The  interlaboratory  consistency  ("interlaboratory  precision") 
of  the  laboratory  groups  can  be  exhibited  as  follows,  where  groups  joined 
by  the  same  line  did  not  exhibit  significantly  different  precision 
at  the  901  confidence  level. 


Rank  Order 

Laboratory  Group 

Most  Precise 

Medicare  Independent 

Interstate 

JCAH 

AAFP/ASIM 

Least  Precise 

Medicare  Hospital 

The  techniques  used  had  a  considerable  effect  on  the  accuracy  and 
precision  of  reported  analyses.    Table  1  lists  the  techniques  which 
were  most  satisfactorily  applied  and  the  percentage  of  the  participating 
laboratories  which  applied  each  technique.    In  most  instances,  automated 
methods  were  applied  with  equal  or  better  average  accuracy  and  considerably 
better  precision  than  the  corresponding  manual  methods.    Results  reported  by 
laboratories  using  diagnostic  kits  were  consistently  less  precise  than 
other  determinations. 
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Insufficient  information  was  available  to  assess  the  medical  usefulness 
of  the  total  bilirubin  determinations.  Of  the  remaining  seven  constituents, 
only  cholesterol  was  analyzed  by  the  study  participants -with  sufficient 
precision  to  permit  the  interlaboratory  monitoring  over  time  of  the 
variation  in  an  individual  patient's  constituent  concentrations.  In 
contrast,  reference  laboratory  analyses  of  cholesterol,  uric  acid,  urea 
nitrogen,  sodium  and  total  protein  were  all  sufficiently  precise  to 
permit  interlaboratory  monitoring  of  individual  variation.    Those  partici- 
pating laboratories  using  the  best  applied  techniques  also  achieved 
acceptable  interlaboratory  precision  in  analyses  of  these  five  constituents. 

Results  -  Hematology 

As  with  clinical  chemistry,  the  average  laboratory  results  obtained 
by  the  participating  groups  did  not  differ  significantly  at  the  951 
confidence  level.    The  interlaboratory  precision  of  the  laboratory  groups 
can  be  exhibited  as  follows  where  groups  joined  by  the  same  line  did  not 
exhibit  significant  differences  at  the  90%  confidence  level. 


Rank  Order 

Laboratory  Group 

Most  Precise 

Medicare  Independent 

JCAH 

Interstate 

Medicare  Hospital 

Least  Precise 

AAFP/ASIM 

Table  2  shows  the  best  applied  techniques  and  the  percentages  of  partici- 
pating laboratories  using  these  techniques. 

Results  -  Microbiology^ 

The  performance  of  the  Interstate  group  was  significantly  better 
than  the  performance  of  the  other  groups  at  the  95%  confidence  level; 
7.6%  of  the  Interstate  laboratory  determinations  were  incorrect  while 
19.9%  of  all  other  determinations  were  incorrect.    However,  even  a  7.6% 
mis- identification  rate  is  not  satisfactory.    Most  troublesome  are  such 
mis- identifications  as  Neisseria  NOS,  N.  gonorrhoeae  or  N.  meningitidis 
for  the  pure  culture  of  Streptococcus  faecalis. 

The  relative  perfomance  of  the  laboratory  groups  can  be  portrayed 
as  shown,  where  the  lines  join  groups  whose  performance  did  not  differ 
significantly  at  the  95%  confidence  level. 
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Rank  Order 


Laboratory  Group 


Best  Performance 


Interstate 
JCAH 


Worst  Performance 


Medicare  Independent 
AAFP/ASIM 
Medicare  Hospital 


Conclusions 

The  data  indicate  that  high  volume  laboratories  may  be  more  proficient 
than  smaller  laboratories,  such  as  those  which  serve  Doctors'  Offices  and 
Medicare  Certified  Hospitals.    In  microbiology,  7.6%  of  the  Interstate 
laboratory  determinations  were  incorrect,  while  16.5%  of  the  determinations 
by  other  large  laboratories  (JCAH  and  Medicare  Independents)  were  incorrect. 
Thus,  it  would  appear  that  the  CDC  proficiency  testing  program  has 
considerably  improved  the  microbiology  performance  of  the  enrolled  labora- 
tories.   Conversely,  clinical  chemistry  and  hematology  analyses  by  the 
Interstate  laboratories  were  no  better  than  comparable  analyses  by  other 
large  laboratories,  many  of  whom  do  not  engage  in  routine  proficiency 
testing  programs.    This  seems  to  indicate  that  the  CDC  proficiency  testing 
programs  in  clinical  chemistry  and  hematology  have  had  relatively  little 
effect  upon  the  performance  of  laboratories  participating  in  the  program. 
This  conclusion  is  further  substantiated  in  a  companion  report.*    It  is 
particularly  important  to  improve  the  effectiveness  of  these  programs 
because  the  interlaboratory  consistency  of  study  participants  with 
respect  to  clinical  chemistry  and  hematology  was  too  often  insufficient 
to  support  monitoring  of  an  individual's  constituent  concentrations  over 
time.    It  appears  that  poor  selection  of  techniques  is  an  important  factor 
in  the  low  rate  of  acceptability  of  laboratory  determinations. 


Limitations 

It  must  be  clearly  understood  that  the  results  of  this  survey  are 
limited  by  four  important  considerations: 

1.  Because  all  of  the  laboratories  participated  on  a  purely  voluntary 
basis,  no  straightforward  extrapolation  can  be  made  to  the  larger  universe 
of  unsampled  clinical  laboratories. 

2.  It  is  probable  that  the  results  of  this  study  do  not  represent 
routine  laboratory  performance  for  two  reasons:     (a)  a  laboratory  probably 
would  not  volunteer  if  its  management  felt  that  to  do  so  would  be 
disadvantageous,  and  (h)  the  sample  materials  probably  received  special 
attention  in  many  of  the  smaller  laboratories  which  were  unfamiliar  with 
analyzing  proficiency  test  samples. 


*Clinical  Laboratory  Performance  Analysis  Using  Proficiency  Test  Statistics, 
NBS  Report,  NBSIR  73  197,  1973  (to  be  published). 
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3.  The  clinical  chemistry  test  specimens  were  prepared  by  a 
dialyzation  process  which  removes  naturally  occurring  reducing  agents 
and  other  substances.    As  a  result,  the  accuracy  of  some  methods,  as 
applied  to  the  test  specimens,  might  differ  from  their  accuracy  in 
analyses  of  human  serum. 

4.  The  true  constituent  concentrations  of  cholesterol  and  the 
hematology  constituents  could  not  be  exactly  determined.    For  these 
constituents  accuracy  was  assessed  relative  to  the  mean  reference 
laboratory  assays. 


Recommendat  ions 

Satisfactory  performance  in  a  microbiology  proficiency  testing 
program  conducted  under  the  auspices  of  either  Federal  or  other  approved 
authorities  should  be  a  legislative  requirement  for  all  clinical  labora- 
tories analyzing  microbiological  specimens. 

A  Technical  Advisory  Committee  consisting  of  government  and  profes- 
sional society  representatives  should  be  established  to  identify  the  most 
accurate  and  precise  analytical  methods  available  and  encourage  their 
use  by  the  largest  possible  number  of  clinical  laboratories.    Zones  of 
acceptable  performance  for  proficiency  testing  should  be  constructed  in 
a  manner  which  reflects  the  variability  associated  with  the  more  accurate 
and  precise  methods  and  systems.    In  this  way,  failure  to  accept  the 
recommended  procedures  would  increase  the  risk  of  unacceptable  performance 
ratings . 

An  experimental  study  should  be  undertaken  to  determine  a  better 
design  for  proficiency  testing  programs  in  clinical  chemistry  and  hematology. 
An  empirical  description  of  the  causes  of  inadequate  laboratory  work  should 
be  used  in  defining  alternative  testing  strategies  for  consideration.  This 
study  should  deal  with  such  questions  as  frequency  of  sair^ling;  feedback 
to  participants;  number  of  levels  at  \^^ich  to  test;  long-term  monitoring 
of  intralaboratory  variability;  follow-up  procedures  on  outlier  values; 
and  the  criteria  for  scoring,  ranking  or  rating  laboratory  performance 
and  its  medical  usefulness.    Until  the  results  of  this  recommended  study 
become  available,  it  does  not  appear  justified  (or  warranted)  to  alter 
the  frequency  of  CDC  proficiency  testing  in  clinical  chemistry  and 
hematology. 


vii 


PORM  NBS-n4A  (1-71) 


U.S.  DEPT.  OF  COMM.           1 .  PU  B  LIC  AT  ION  OR  R  E  PORT  NO.           2.  Gov't  Accession 

BIBLIOGRAPHIC  DATA                        mhotd   7-5    i  AO 
5l^£gj                                NBSIR  7J-iDZ 

3.  Recipient's  Accession  No. 

4.  TITLE  AND  SUBTITLE 

Executive  Summary  of  A  Proficiency  Test  Assessment  of  Clinical 
Laboratory  Capability  in  the  United  States 

5.   Publication  Date 

6.  Performing  Organization  Code 

7.  AUTHOR(S) 

Peter  W.  Finkel  and  Ted  R.  Miller 

8.  Performing  Organization 

NBSIR  73-162 

9.  PERFORMING  ORGANIZATION  NAME  AND  ADDRESS 

NATIONAL  BUREAU  OF  STANDARDS 
UEFAKIMENI  Or  CUMMhRCb 
WASHINGTON,  D.C.  20234 

10.  Project/Task/Work  Unit  No. 

4314457 

11.  Contract/Grant  No. 

12.  Sponsoring  Organization  Name  and  Address 

Division  of  Health  Evaluation 

Office  of  the  Assistant  Secretary  for  Planning,  and  Evaluation 

Department  of  Health,  Education  and  Welfare 

Room  5526  HEW  North  Building,  Washington,  D.  C.  20201 

13.  Type  of  Report  &  Period 
Covered 

Final  3/71  -  4/73 

14.  Sponsoring  Agency  Code 

15.  SUPPLEMENTARY  NOTES 


16.  ABSTRACT  (A  200-word  or  less  factual  summary  of  most  significant  information.   If  document  includes  a  significant 
bibliography  or  literature  survey,  mention  it  here.) 

Executive  summary  of  a  report  abstracted  as  follows: 

The  proficiency  of  a  selected  sample  of  physician,  hospital  and  independent 
laboratories  was  assessed  with  respect  to  their  ability  to  analyze  clinical  chemistry 
and  hematology  samples  and  to  identify  microbiological  organisms.    For  the  assessment 
of  clinical  chemistry  and  hematology  proficiency,  the  laboratories  were  grouped  and 
determinations  of  group  accuracy  and  group  precision  were  made.    Further  analyses 
were  performed  to  determine  relative  accuracy  and  precision  of  the  techniques 
presently  applied  by  these  groups.    There  was  no  significant  difference  at  the  951 
confidence  level  in  the  accuracy  achieved  by  the  various  laboratory  groups  involved  in 
Clinical  chemistry  and  hematology  analysis.     In  clinical  chemistry,  the  Medicare- 
Certified  Independent  Laboratories,  CDC  Tested  Laboratories  and  JCAH-Members  generally 
proved  more  precise  than  Physicians'  Office  and  Medicare- Certified  Hospital  Laboratories. 
However,  none  of  the  laboratory  groups  were  sufficiently  accurate  to  permit  the 
monitoring  over  time  of  variation  in  an  individual  patient's  constituent  concentrationf; . 
It  would  appear  that  poor  selection  of  techniques  was  an  important  contributor  to  this 
low  performance  level.    In  hematology  the  Physicians'  Office  Laboratories  proved  to  be 
the  least  precise  of  the  groups.    There  was  no  noticeable  difference  in  precision 
between  participants  in  the  CDC  proficiency  testing  program  and  non-participants.  With 
respect  to  microbiology,  7.6%  of  the  identifications  by  laboratories  participating  in 
the  CDC  testing  program  were  incorrect,  while  19.4%  of  all  other  identifications 
were  incorrect. 


'    17.  KEY  WORDS  (Alphabetical  order,  separated  by  semicolons) 


i|     Accuracy;  clinical  chemistry;  hematology;  medical  usefulness;  microbiology,  proficienc)' 


,  r.estmg  —  1 

18.  AVaILA^ILLi  1  STATEMEiMx 

19.  SECURITY  CLASS 
(THIS  REPORT) 

21.  NO.  OF  PAGES 

UNLIMITED. 

UNCL  ASSIFIED 

1     1  FOR  OFFICIAL  DISTRIBUTION.   DO  NOT  RELEASE 
TO  NTIS. 

20.  SECURITY  CLASS 
(THIS  PAGE) 

UNCLASSIFIED 

22.  Price 

USCOMM-DC  66244-P71 


